12. Draw Conclusions

Re: Analyze Data

For the test of the invariant metric, number of cookies, there were a larger
number of cookies recorded in the experiment group, 47 346 vs. 46 851. This
ends up generating a p-value of 0.107 (z = -1.61), which is within a
reasonable range under the null hypothesis. Since we lack sufficient reason to
reject the null, we can continue on to evaluating the evaluation metrics.
(Note that this doesn't mean that there wasn't something actually different
about the cookie counts between groups, only that we couldn't detect it if such
a difference existed.)

For the first evaluation metric, download rate, there was an extremely
convincing effect. An absolute increase from 0.1612 to 0.1805 results in a
z-score of 7.87, well beyond any standard significance bound. However, the
second evaluation metric, license purchasing rate, only shows a small increase
from 0.0210 to 0.0213 (following the assumption that only the first 21 days of
cookies account for all purchases). This results in a p-value of 0.398
(z = 0.26).